Schierhout et ol. Implementation Science 2013, 8:1 19 
http://www.implementationscience.eom/content/8/1/1 19 IMPLEMENTATION SCIENCE 



Implementation 
Science 



RESEARCH Open Access 



Evaluating the effectiveness of a multifaceted, 
multilevel continuous quality improvement 
program in primary health care: developing a 
realist theory of change 

Gill Schierhout 1 *, Jennifer Hains 1 , Damin Si 2 , Catherine Kennedy 3 , Rhonda Cox 4 , Ru Kwedza 2 , 

Lynette O'Donoghue 5 , Marea Fittock 5 , Jenny Brands 1 , Katherine Lonergan 1 , Michelle Dowden 6 and Ross Bailie 1 



Abstract 

Background: Variation in effectiveness of continuous quality improvement (CQI) interventions between services is 
commonly reported, but with little explanation of how contextual and other factors may interact to produce this 
variation. Therefore, there is scant information available on which policy makers can draw to inform effective 
implementation in different settings. In this paper, we explore how patterns of change in delivery of services may 
have been achieved in a diverse range of health centers participating in a wide-scale program to achieve 
improvements in quality of care for Indigenous Australians. 

Methods: We elicited key informants' interpretations of factors explaining patterns of change in delivery of 
guideline-scheduled services over three or more years of a wide-scale CQI project, and inductively analyzed these 
interpretations to propose fine-grained realist hypotheses about what works for whom and in what circumstances. 
Data were derived from annual clinical audits from 36 health centers operating in diverse settings, quarterly project 
monitoring reports, and workshops with 12 key informants who had key roles in project implementation. We 
abstracted potential context-mechanism-outcome configurations from the data, and based on these, identified 
potential program-strengthening strategies. 

Results: Several context-specific, mechanism-based explanations for effectiveness of this CQI project were 
identified. These were collective valuing of clinical data for improvement purposes; collective efficacy; and 
organizational change towards a population health orientation. Health centers with strong central management of 
CQI, and those in which CQI efforts were more dependent on local health center initiative and were adapted to 
resonate with local priorities were both favorable contexts for collective valuing of clinical data. Where health 
centers had prior positive experiences of collaboration, effects appeared to be achieved at least partly through the 
mechanism of collective efficacy. Strong community linkages, staff ability to identify with patients, and staff having 
the skills and support to take broad ranging action, were favorable contexts for the mechanism of increased 
population health orientation. 

Conclusions: Our study provides evidence to support strategies for program strengthening described in the 
literature, and extends the understanding of mechanisms through which strategies may be effective in achieving 
particular outcomes in different contexts. 
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Background 

Over the past ten years the Audit and Best Practice for 
Chronic Disease Project (the ABCD project) has devel- 
oped and supported quality improvement tools and pro- 
cesses in primary health care centers across Australia, 
with a focus on centers that serve predominantly Indi- 
genous populations. We have previously reported on en- 
ablers and barriers to participation in the project by 
health centers, with the objective of informing the devel- 
opment of more effective strategies for supporting up- 
take [1]. In this paper, our focus is on describing how 
contextual and other factors may interact to influence ser- 
vice delivery outcomes, particularly the desired outcome 
shared by most continuous quality improvement (CQI) 
initiatives, that of improved delivery of recommended care 
processes. 

Variation in the effectiveness of CQI interventions has 
been commonly reported [2], and a range of theories, 
models, and empirical studies have advanced the under- 
standing of this variation. However, there is a lack of un- 
derstanding about how the various drivers of CQI 
effectiveness that have been identified in the health con- 
text interact with one another and with contextual fac- 
tors to achieve desired outcomes [3]. There are few 
published empirical studies in this area from primary 
health care settings, and little clear guidance to policy 
makers or planners regarding how and in which circum- 
stances CQI interventions could be modified to strengthen 
desired impacts in primary health care [3,4]. Our study 
makes an effort to address this gap. 

The ABCD Project was designed to support best prac- 
tice in prevention and management of chronic disease in 
Indigenous primary health care services in Australia. 
There are many different CQI implementation models 
described in the literature, with the ABCD CQI project 
sharing characteristics with what has been termed 'inte- 
grated CQI', that is, CQI models that are multi-site and 
multi-faceted and aim to achieve change at various levels 
of the system [5,6]. Between 2002 and 2006, the project 
used participatory action research methods at the 
organizational level to introduce and examine the impact 
of CQI in 12 remote Indigenous primary health care 
centers [7-9]. The extension phase of the project (2005 
to 2009) was designed to examine the operational and 
policy requirements of expanding the ABCD model to 
other geographic locations, and to other core compo- 
nents of primary health care, including maternal and 
child health and mental health [10]. The project design 
drew on diffusion of innovation theory [11] (the 'how') 
and on CQI theory and methods (the 'what'). By the end 
of 2009, the tools and processes developed through the 
project had been used in about 130 Indigenous primary 
health care centers across the country, including 69 ser- 
vices which were formally enrolled in the project 



research activities. Ethics approval was obtained from re- 
search ethics committees in each jurisdiction. 

At the health center level, health centers were sup- 
ported to conduct annual quality improvement cycles 
(plan-do-study-act) and a web-based information system 
provided participants with real-time analysis of their per- 
formance data, and capacity to compare it with others in 
their region and across the project. Health center staff 
were supported through their annual CQI cycles by 
state/territory co-ordinators (referred to as 'hub co- 
ordinators') appointed for this purpose. At the regional 
level, hub co-ordinators based in each of the five regions 
where the project was operating, had links to regional level 
management and academic support. In some regions, 
health boards, or umbrella organizations managing a group 
of health centers, took on implementation co-ordination 
roles, including sharing of lessons between organizations 
within their group. At the national level, the project core, 
based in an academic institution, undertook ongoing re- 
finements to project design to ensure consistency with na- 
tional and international evidence-based guidelines, and 
hosted annual feedback and planning meetings to which all 
participating health centers were invited. A number of pro- 
ject staff and affiliated researchers were engaged with the 
development of national and jurisdictional chronic disease 
policy [12]. Previous project publications have described 
how the diversity of health centers participating in the 
ABCD project took up the project in different ways [1], 
and the considerable variation in their delivery of services 
that are scheduled at specific intervals according to best 
practice guidelines ('guideline-scheduled services') [7-9]. 

In this paper, we combine inductive analytic ap- 
proaches with principles of realist evaluation [13] to 
identify fine-grained hypotheses about how, and in what 
contexts, the ABCD CQI project may have achieved 
changes in delivery of guideline-scheduled services, fo- 
cusing on services for diabetes and preventive care. The 
overall aim of the study is to identify how, why, and in 
what contexts CQI in primary health care may achieve 
various outcomes — and to share findings of relevance to 
implementation researchers, managers, and planners. 

Methods 

Research strategy 

We elicited the interpretations of key informants regard- 
ing the factors explaining patterns of change in delivery 
of guideline-scheduled services during implementation 
of a wide-scale CQI project, and analyzed these interpre- 
tations in order to propose fine-grained realist hypotheses 
about what works for whom and in what circumstances. 
Theoretical explanations of this kind, or 'middle-range 
theories' '...involve abstraction... but [are] close enough to 
observed data to be incorporated in propositions that per- 
mit empirical testing' [14]. 
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Realist evaluation starts with program theory, uses the- 
ory to guide data collection and analysis, and ends with 
refined program theory. The program theory in realist 
evaluation specifies a relationship between context, 
mechanism, and outcome — or 'CMO' configuration 
(CMO)' [13]. A key methodological challenge in apply- 
ing realist evaluation in health systems research is identi- 
fying middle-range theory [13,15]. Our research strategy 
of developing middle-range theory based on program 
implementers' views is consistent with recommendations 
from a recent overview of the use of realist evaluation 
methods in health systems research, which proposes this 
as an explicit additional step in situations where there is 
little to inform a-priori development of an appropriate 
middle-range theory [15]. The specific research objective 
of the study was to empirically identify CMO configura- 
tions or 'middle range theory' that may explain the ef- 
fectiveness of CQI at health center level. 

Data sources and analysis 

For the purposes of developing CMO configurations, or 
middle-range theory, we defined outcomes at the micro- 
system (health center level) as the starting point of the 
enquiry. There were 36 health centers across the States/ 
Territories that had participated in the ABCD CQI pro- 
ject for three or more full annual QI cycles. As the 



analysis presented in this paper is concerned with 
achievement of improvements in quality of care over 
time, the study focused on these 36 health centers. 
These included both large and small health centers, in 
remote, regional, and urban locations. The data sources 
that were drawn on, and the approach followed is 
presented diagrammatically in Figure 1, and outlined 
below. 

Hub co-ordinators responsible for supporting health 
centers with implementation, provided regular quarterly 
reports to the project management committee on imple- 
mentation progress (16 to 18 reports over four to five 
years for each hub coordinator; 'Routine program 
reporting' in Figure 1), and participated in a series of 
workshops over the course of the project (eight workshops 
over four years; 'Workshops with hub-coordinators' in 
Figure 1). In these workshops, the hub coordinators, the 
project manager, and key investigators worked to define 
the salient factors for effective implementation of the 
ABCD CQI project based on their experience, and in rela- 
tion to factors identified in the literature. As a result of 
their experience on this project and their more general ex- 
perience of working with services involved in the project, 
many of the workshop participants had insight into the 
day-to-day operations of many of the participating health 
centers. Through the workshop process we were thus able 
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Figure 1 Outline of data sources and study process. 
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to compare and contrast the insights of various workshop 
participants into factors that were present across different 
health centers that showed similar patterns in their data. 
The analysis was therefore not dependent only on the in- 
sights of one individual into the circumstances of each 
health center. The workshop process was complemented 
by ongoing liaison between the hub co-ordinators, key in- 
vestigators, and the project manager in iterative cycles of 
clarification of concepts and refinement of the analysis. 

Selection and presentation of outcome patterns 

Drawing on clinical audit data collected by each health 
center as part of their CQI cycles, we constructed a 
measure of overall delivery of scheduled services for dia- 
betes and for preventive care to well adults for each 
health center for each year of participation. Quantitative 
methods to examine variation on these measures across 
the project have been previously reported [7,8]. From 
the diverse range of patterns of change in the clinical 
audit data for each health center over the duration of 
the study, we identified a number of major qualitative 
patterns of change (along the lines of the data presented 
in Figure 2). We did this separately for patterns of 
change in delivery of diabetes-related services and for 
delivery of preventive services ('Patterns of change in 
overall delivery of care for diabetes/preventive care' in 
Figure 2). It was inevitable that some health centers did 
not show patterns as clearly as others, and that for some 
health centers, respondents were less able to identify ex- 
planation for patterns of change. The selection of trend 
lines shown in Figure 2 are for health centers that 
showed trends in delivery of care that were most clearly il- 
lustrative of the patterns that were discussed, and those 
for which we had the best explanatory data. Further details 
on how these measures were constructed and patterns of 
change identified, are provided in Additional file 1. 

Key informants perspectives on factors influencing 
outcome patterns 

Late in the final year of the extension phase of the ABCD 
project, graphs showing changes over time in the audit 
data for each health center, categorized by the major quali- 
tative patterns of change, were presented and discussed in 
a workshop with the regional hub-co-ordinators, key 
investigators and other members of the project man- 
agement committee ('two day facilitated workshop' in 
Figure 1). These 12 key informants had detailed insight 
into the experience of engagement by health centers in 
the CQI process through their involvement in supporting 
local and regional level implementation, and had access to 
quarterly progress reports for the three years of project 
implementation. During the workshop discussions, infor- 
mants were asked to consider the situation of each health 
center over the time covered by the study, and give their 



perceptions on what contributed to the patterns of change 
in delivery of services. Discussions were recorded and 
transcribed verbatim so that the raw data could be system- 
atically analyzed. 

Analysis of the key informants' perspectives 

As a first step a preliminary thematic summary of key infor- 
mants' interpretation of factors explaining outcome pat- 
terns was produced ('Preliminary thematic analysis' in 
Figure 1). This thematic summary was based on project im- 
plementers' understanding of what could explain the vari- 
ous outcome patterns observed in the quantitative data. 
Members of the study team met to discuss the emerging 
themes, and the thematic summary was circulated and re- 
fined on the basis of feedback and discussion using a group 
consensus approach to strengthen the validity of our find- 
ings. Broad preliminary themes that were identified in- 
cluded: data and data systems; regional support; leadership 
and management; organizational culture; approach to ser- 
vice delivery; community linkages; adequacy and stability of 
staffing; and ownership and management of CQI. Following 
this, the themes were refined and categorized into 'mechan- 
ism', 'context' or 'intermediate outcome', on the basis of the 
realist conception of mechanism and outcomes [13], and 
drawing on the typology of mechanisms in the context of 
program evaluation outlined by Astbury and Leuuw [14] 
(candidate mechanisms and contexts' in Figure 1). In iden- 
tifying mechanisms and the contexts that enabled them, we 
drew on organizational theories of change [16] and theories 
and models of quality improvement [17]. We drew only on 
aspects of these theories and models that could be identi- 
fied or grounded in our qualitative data and were consistent 
with program logic and activities. 

Through an iterative process of checking data against 
candidate CMO propositions, we sought to identify and 
refine key mechanisms that appeared to best explain 
health center level changes in delivery of guideline sched- 
uled services over time ('outcomes') and what contexts 
seemed to enable or inhibit these mechanisms ('checking 
for vulnerabilities and contradictions' in Figure 1). The 
initial CMO configurations were redefined and refined 
through a process of looking across the different patterns 
of change, at 'positive' and 'negative' cases, and where the 
same mechanism appeared to be enabled by different con- 
texts. We attempted as far as possible to include all of the 
common factors that were identified as explaining each of 
the outcome patterns. Based on transparent reasoning, we 
identified specific strategies for strengthening wide-scale 
CQI programs in identified contexts, which may merit 
further testing and refinement. 

Results and discussion 

Data presented in Figure 2 demonstrate wide variation 
in baseline level of performance and divergent patterns 
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of change achieved by health centers for overall diabetes 
and preventive care. As will be evident in the presenta- 
tion of results below, the participating health centers 
also showed considerable heterogeneity in their contexts, 
activities, and outcomes. 

We identified three main mechanisms, and seven po- 
tential CMO configurations as candidates to explain dif- 
ferent patterns of change. Table 1 shows these CMO 
configurations, together with exemplar quotes, and a 
summary of the specific inputs of the ABCD CQI pro- 
ject relevant to each of the proposed mechanisms. In the 
text below, we describe each of the mechanisms; what 
contexts appeared to trigger or inhibit each mechanism; 
and what outcome patterns resulted. Specific health cen- 
ter examples are presented in order to illustrate our rea- 
soning. The numbering of the trend lines shown in 
Figure 2 allows linking of the textual explanation of fac- 
tors underlying various patterns of change to the trend 
lines for specific health centers. In the presentation of 
results below, where contexts are discussed they are 
numbered in order of citation and referred to as CI, 
C2.... A similar approach was followed in discussion of 
mechanisms (Ml, M2...). The CMO configurations are 
reframed into proposition statements, encapsulating 
the most important findings of use to researchers, 
managers, and practitioners (see section "List of CMO 
configurations reframed as proposition statements"). 
Potential strategies for strengthening wide scale CQI 
projects in different primary health care contexts and 
the lines of reasoning that led to them are provided in 
Table 2. 

List of CMO configurations reframed as proposition 
statements encapsulating the most important key findings 

1. The mechanism of collective or shared valuing of 
clinical data for improvement purposes (Ml) can be 
enabled by the context either of centralized 
management style of a regional board, where CQI 
and change is centrally-led (CI), or in situations 
where implementation of CQI is devolved to local 
health center level, providing that local health 
centers have competent and experienced staff with 
capacity to appreciate potential of data to improve 
clinical care (C2). Staff capacity and co-operation 
are likely to be key constraints on this mechanism, 
particularly in situations of relative isolation of 
services and small staff complements (C3). 
Further, this mechanism may also be enabled at 
the macro level. Where this occurs, its influence 
at a health center level may be more important in 
the context of local devolution of CQI 
implementation (C2) compared to centrally-led 
implementation (CI). 



2. Collective change efficacy as a mechanism (M2) can 
be enabled by the existence of regional or 
organizational infrastructure that supports 
networking for CQI, particularly where there is 
positive prior history of collaboration (C4). Remote 
and geographic dispersion of health centers may be 
an additional favorable contextual factor for change 
efficacy as networking type activities between health 
centers may be more valued in situations of relative 
isolation. Competitiveness and organizational culture 
unsupportive of collaboration where there is role 
confusion and/or poor co-ordination between 
service providers (C5) is likely to inhibit activation 
of this mechanism. 

3. Organizational change to encompass a population 
health orientation (M3) can be enabled by the 
context of stable effective outreach staff and good 
regional co-ordination (C6), and by the contexts in 
which health center staff identify with patients, and 
have leadership skills to take broad ranging action in 
CQI (C7). Resource constraints and financial 
incentive structures may act as moderating 
influences on the outcome patterns achieved. 

Mechanism 1: Collective valuing of clinical data for 
performance improvement 

Our analysis suggested that one of the processes through 
which improvement came about was through a collect- 
ive or shared valuing of clinical data for improvement 
purposes. This was expressed through the attitudes of 
health service staff and managers towards clinical data. 
For example, in explaining marked change to improve- 
ment, and sustained high performance, informants spoke 
about individual health center staff who were passionate 
and committed to using clinical data to improve service 
delivery, along with organizational initiatives to develop 
and improve the capability of clinical information sys- 
tems to provide data for this purpose (Table 1, exemplar 
quotes). 

Our analysis identified three main contexts that 
appeared to trigger (or inhibit) this mechanism (Ml). 
These are outlined below. 

CI: Centralized management style; regional board 
committed and involved in CQI implementation 

One of the challenges faced by health centers implementing 
the ABCD CQI project was the inconsistent state of 
development of clinical record keeping systems, and 
the constraints this imposed on their ability to collate 
adequate data on care processes through clinical audit 
[1]. Some health centers undertook major revisions of 
their clinical record keeping systems in response to the 
need highlighted by the project. Although improve- 
ments in clinical record keeping occurred in a number 



Table 1 Proposed CMO configurations explaining how a wide-scale CQI model in primary health influences care delivery 



Summary of salient ABCD CQI project 
inputs 



Potential contexts 



Plausible 
mechanisms 



Potential outcomes 



Exemplar quotes 



• Health center staff participate in annual 
predominantly paper-based audit processes, 
interpretation of reports and systems 
assessment and action planning that use 
data derived from clinical audit, as a starting 
point for change 



• Automatic generation of reports from 
clinical audit through a web-based 
information system. 



• Ability for health centers to adjust 
reporting (format, indicators etc.) to suit 
local reporting requirements and 
accountabilities 



• Engagement of champions and change 
agents at different levels of the health 
system to promote uptake of the project 

• Ongoing refinement of the project to 
maximize synergies with major policy 
initiatives 



• Processes and tools that brought 
together different health care professionals 
and managers to share ideas for service 
performance and improvement activities 



• Annual planning meetings, meetings, 
teleconferences and sharing of experiences 
between health centers 



Centralized management style; regional 
board committed and involved in CQI 
implementation (CI) 



Local ownership of CQI (devolved 
management style); competent staff in 
management roles; managers and 
clinicians with an interest in chronic 
disease and in clinical and population 
health data (C2) 



Poor management, uncertainty and 
confusion over role definitions. (C3) 



Regional or organizational infrastructure 
supportive of networking for CQI and 
centralization of some tasks. Positive prior 
history of collaboration (C4) 



Organizational culture unsupportive of 
collaboration. Health centers see 
themselves as being in competition (C5) 



Collective or • Temporary declines and instability as 
shared valuing services get used to new systems. Major 
of clinical data revision of clinical record keeping; 
for improvement centralized 'cleaning up' process to 
purposes (M1) standardize reporting across health 
centers. 

• Marked changes (HC1 & 2 Figure 2E). 



Collective or • Use of non-core strategies such as 

shared valuing follow up of individuals receiving poor 

of clinical data care identified in clinical audit, used to 

for improvement highlight clinical relevance of data (HC 7 & 

purposes (M1) 8 Figure 2A). 

• Sustained high performance or marked 
change to improvement (HC 3 Figure 2C I 
2F). 



Collective or 
shared valuing 
of clinical data 
for improvement 
purposes (M1) 



• Limited changes in data systems; 
frustration and confusion about ongoing 
involvement in CQI. 

• Poor performance or declines in care 
(HC5 & 6 Figure 2C and 2D). 



Collective 
change efficacy 
(M2) 



Collective 
change efficacy 
(M2) 



• Appropriate reflection on salient 
comparison group; formation of 
networked communities 

• Sustained high performance or marked 
change to improvement (HC3 Figure 2F 
and 2C). 

• Inappropriate reflection on performance 
and early fatigue 



'Across our region we did a concerted effort 
for documentation for diabetes services... 
and so certainly the improvement [in the 
early years] would just have been about 
documentation, so having somewhere to 
write things... I think all was about 
documentation. But 2006 to 2007 I think 
there was a concerted effort. The chronic 
disease strategy really kicked in and that 
was when, at some point during that 
period, [name] had her lights on moment, 
[we understood] the focus of how 
important doing the right processes at the 
right time was'. 

'Doctor [name] was always really, really 
interested in the data... where he saw really 
big increases in ACRs and that, he would 
want to know who were the people that 
were being audited in terms of following 
those up. So he was very good with that. 
And then of course [name] is their quality 
improvement person... they were 
standardising their filing system right across 
that region, which she led, and so [the 
data] were quite easily accessible' 

'A lot of health workers. Been there for a 
long time, and I asked them what, sort of 
asked what their training was. Why aren't 
they doing like blood pressures and blood 
sugars. . They said they were not allowed. 
They've been told by management they're 
not allowed.. . .that was part of their training 
though that, you know, I'm a health worker 
and that's part of my training. But yeah, a 
lot of them have been there for 15 years. 
They just didn't have a focus. We actually 
wondered what their existence was about'. 

'Have good communication systems... 
share ideas between the different health 
centers. And a strong focus on education 
through regional support teams. . . use 
video conferencing as well as regular visits., 
and its very vibrant, like they are always out 
there'. 

'Cause, yeah, when I first started they were 
really eager, you know, like doctors were all 
eager to see how, cause there's three clinics 



Table 1 Proposed CMO configurations explaining how a wide-scale CQI model in primary health influences care delivery (Continued) 



• Provision of benchmarking data, allowing 
health centers to reflect on their 
performance in relation to that of others 

• Application of CQI to a wide range of 
health outcomes and service populations 
(diabetes, preventive health, maternal 
health, child health), and a range of care 
processes 

• Processes that brought different service 
delivery professionals together to reflect on 
health center performance (for example, 
outreach workers and clinic-based staff) 



• Regionally based co-ordinator positions 
supported population health planning and 
multidisciplinary team approaches to 
chronic disease care 



Pre-existing favorable context of patient 
and community oriented care, supported 
by stable effective outreach workers and 
good regional co-ordination for CQI (C6) 



Staff who can identify with patients and 
have the skills to take broad ranging 
action, including clinical action and action 
related to data system development and 
use, coupled by regional support and co- 
ordination (C7) 



Organizational 
change to 
encompass a 
population 
health 

orientation (M3) 



Organizational 
change to 
encompass a 
population 
health 

orientation (M3) 



• Persistent low performance or declines 
in care (HC5 and HC 6, Figure 2C and 2D). 



• Recognition of value and roles of 
Aboriginal Health Workers in outreach and 
linking this to service delivery. 

• Developing greater consistency in 
provision of general practitioner services. 

• High performance and marked change 
to improvement (HC10 Figure 2C and 2F). 



• Priority-driven resource allocation 
decisions. 

• Mixed patterns (high performance or 
marked change to improvement in 
diabetes and low in prevention or vice 
versa) (HC12 Figure 2B and 2F and HC13 
Figure 2C and 2D). 



in [name of city]. They were all competing 
with each other, who's going to be the 
best, and who's going to give the best 
service, so but it's just worn off. 

'With [NAME] they had the self 
management program there, and they get 
a lot of stuff outside the health center., it 
was about promoting good health in the 
community, working with the store [for 
supply of healthy food in this remote 
community], and those places. A lot of 
health promotion activities were going on 
with those health workers there. .. 
Population lists were being improved and a 
better understanding [in the context of 
transient populations and population 
movement]'. 

'PI: Well [NAME] is passionate about 
making sure all the diabetics [are well cared 
for] ...P2: He was also a diabetic wasn't he? 
P1: Yeah. He had a personal drive and he 
was cardiac nurse, so any cardiac stuff that 
was related to diabetes, you know, he could 
tell people when they were being sent to 
Adelaide and you know, he did all that sort 
of advice as well. . .And what he did though 
was set up the big clean up of the data 
system. And started extracting reports and 
cleaning up the population base'. 
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Table 2 Potential strategies for strengthening wide-scale CQI projects to enhance clinical performance in different 
contexts 



Context 



Proposed mechanisms and reasoning for recommended strategies 



Mechanism 1: Collective or shared valuing of clinical data for improvement purposes: if health centers expect their clinical audit data to be fit for the 
purpose of Ql, then they will be more motivated to use these data for service improvement as envisaged by the CQI model. 



Centralized management style; regional board committed and involved 
in CQI implementation (CI) 



Local ownership of CQI; competent managers with interest in chronic 
disease and clinical and population health data (C2) 



• If centralized management of CQI institutes revision of clinical record 
keeping systems, participating health centers will develop collective or 
shared valuing of clinical data for improvement purposes, and will in fact 
use the data for performance improvement, resulting in improvements in 
care delivery. 

• If this works because of the expectations of the potential for data to 
support CQI (for example, through social mechanisms such as the 'self 
fulfilling prophecy'), then 

wide-scale CQI projects could encourage health centers sharing this 
context to enter CQI processes with optimism, and use processes as a way 
to motivate for improvements in clinical record keeping even where good 
quality data on care processes are not consistently available at the outset 

• If clinical staff use data in non-core ways to illustrate the applicability of 
data and importance of record keeping, health centers participating in 
these initiatives will develop collective or shared valuing of clinical data for 
improvement purposes, and will in fact use the data for performance 
improvement, resulting in improvements in care delivery. 

• If this works because of the adaptive potential of the project then 

wide-scale CQI projects could develop examples of different presentation 
formats of audit data, and of CQI processes to illustrate adaptive potential 
more strongly, demonstrating their capacity to resonate with different 
organizational cultures and vision 

• If poor overall management and role confusion detracts from health 
center staff perceptions of the value of their data, health centers 
participating in wide scale CQI projects are less likely to develop shared 
valuing of clinical data for improvement, and will be less likely to use the 
data for performance improvement, constraining the potential for 
improvements in care delivery, and discouragement (negative feedback 
loop). 

• If this context is a key constraint on the effectiveness of CQI, then 

interventions targeting unfavorable organizational contexts should be 
developed, prior to, or in parallel with, CQI implementation 

Mechanism 2: Collective efficacy - If health center staff have a strong sense of shared belief of achieving improvement through the CQI project, then 
they will be more motivated to attempt changes to improve service delivery as envisaged by CQI, devote considerable effort to it, and persist in the 
face of difficulties. 



Poor management, uncertainty and confusion over role definitions (C3) 



nfrastructure supportive of CQI networking; positive prior history of 
collaboration (C4) 



Organizational culture unsupportive of collaboration (C5) 



• If regional/organizational infrastructure is supportive of networking for 
CQI, and networks are formed, health centers will attempt changes, put 
effort into changes and show persistence, resulting in improvements in 
care delivery. 

• If this works because of informal social control enacted under conditions 
of social trust, then 

wide-scale CQI projects could encourage greater density of networks 
between health centers in this context, transparent sharing of information 
and experiences related to CQI 

• If organizational culture is unsupportive of collaboration, inappropriate 
competitiveness and early fatigue and disillusionment will result. If this 
'works' because of lack of co-operation with social control, related to lack of 
social trust, then 

wide scale CQI projects could seek to identify health centers sharing this 
context, and aim to build sufficient trust for collaborative networking to 
take place 
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Table 2 Potential strategies for strengthening wide-scale CQI projects to enhance clinical performance in different 
contexts (Continued) 

Mechanism 3: Organizational changes towards a population health orientation - If health centers share an understanding of their role as supporting 
health of their service and community populations, not just those presenting for care, then they will engage in activities outside of the health center, 
build trust with community members, instituting changes for service improvement that are consistent with community needs, and therefore more 
likely to be acceptable to the community and lead to greater demand for services, and increased delivery of guideline scheduled services - as long 
as the guidelines and indicators measured are consistent with community needs. 

Pre-existing favorable context of patient and community oriented care, • If organizational culture has a strong external focus, participation in CQI 
supported by stable effective outreach workers and good regional co- may enable clearer understanding of unmet need/under delivery, helping 
ordination for CQI (C6) health centers to galvanize to improve care, and will use these data for 

performance improvement. 

• If this works because of the role of CQI in providing information on 
population health needs, then 

wide-scale CQI projects could be designed as broad integrated models as 
these will be more likely to trigger change towards a population health 
orientation than narrow CQI models that focus on a more limited range of 
clinical targets 

Staff who can identify with patients and have the skills to take broad • If key individuals are motivated and empowered to take broad ranging 
ranging action (C7) action, and have the support to do so, then they will actively participate in 

wide-scale CQI projects, and use these as a tool to initiate improved care 

delivery 

• If this works because of the role of individual level enthusiasm in 
promoting change, then 

wide-scale CQI projects could seek to proactively build the skills and 
development of enthusiastic clinical leaders in promoting overall 
performance improvement across the scope of clinical care 



of different regions on a limited scale, revision of rec- 
ord keeping systems was most marked in the one re- 
gion where CQI processes were supported by strong 
central management and regional support systems (CI). 
The board in this particular region, responsible for a 
cluster of health centers, implemented a wide ranging 
data quality and service improvement process, includ- 
ing improving clinical information systems and docu- 
mentation of service delivery, record filing, developing 
accurate service population lists and disease registers, 
and implementation of recall and reminder systems. 
For health centers in this region that showed improvements 
in service delivery, key informants identified the attention 
paid by the regional board to improving data systems as an 
important explanatory factor for these patterns of change 
(for example, HC12 and HC14 in Figure 2B). Improved rec- 
ord keeping appeared to be particularly relevant in early 
years of implementation. In later years, patterns of change 
to improvement were explained by capacity and motivation 
of health center staff. 

Several health centers within the same region showed 
marked changes in delivery of services, particularly re- 
lated to diabetes care, as practitioners were learning how 
to use new data systems (Figure 2E, HC1 and HC2). 
Broad ranging efforts to improve documentation and 
data systems sometimes resulted in patterns of improve- 
ment (Ol) and sometimes they contributed to short-or 
medium-term inconsistencies in observed patterns of 
change as clinical record keeping systems were being 
modified and updated (02). 



For two of the health centers that shared this context, 
but that showed declines in service delivery after initial 
improvements (03), staffing constraints appeared to be 
key contextual influences on their ability to sustain 
higher performance. In one of these health centers, a se- 
nior nurse who played a major role in service delivery, 
became disgruntled because of perceived lack of support, 
and refused to work with regional systems supporting best 
practice across the scope of clinical care. This refusal coin- 
cided with loss of a service provided by a visiting endo- 
crinologist — this specialist provider had provided services 
to diabetes clients, and withdrew this service provision 
during the period covered by the year three audits. The 
marked change to decline for this health center was 
explained by the combined effect of these two constraints 
(HC15, Figure 2E and 2F). In another health center 
(HC2), the combined effect of implementing the new data 
system, together with decline in staff capacity as a result 
of a high performing Aboriginal health worker taking on 
part-time study (with consequent less involvement in ser- 
vice delivery), were identified as explaining the marked de- 
cline in performance between years three and four. 

The findings presented above led to the hypothesis 
that one of the ways in which this CQI project achieved 
its effects, was through collective valuing of clinical 
data for improvement purposes, enabled by strong cen- 
tral management of CQI. We noted the role of the so- 
cial situation or event (in this case the regional board 
initiating wide-scale improvements in data systems) in 
shaping the response of individual health centers. 
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Applying the reasoning of the self-fulfilling prophecy 
[18], we hypothesize that collective or shared valuing of 
clinical data for improvement purposes may have 
'worked' to achieve desired outcomes at least partly be- 
cause of the principle that expectations are brought 
about because of the belief that they are justified. Al- 
though there may be rival explanations, this underlying 
social mechanism was a plausible fit with the patterns 
in our data, and suggests that collective valuing of data 
may be strengthened in similar contexts, through strat- 
egies designed to maintain confidence and optimism in 
data (Table 2). 

C2: Local ownership of CQI (devolved management style); 
competent and supportive health center managers and 
clinicians with an interest in chronic disease and in 
clinical and population health data — either at health 
center or regional level 

The presence of influential clinicians who were able to 
relate data to care improvement was commonly cited as 
a factor underlying sustained high performance or 
marked change to improvement. For example, for two of 
the health centers showing these patterns, key infor- 
mants noted that clinicians had been enthusiastic to 
identify and follow up individuals for whom the clinical 
record audit showed gaps in care processes. While the 
follow up of individuals is not a core part of CQI activ- 
ities, these particular clinicians had used this process as 
a means to highlight the value of data and to motivate 
for better clinical record keeping. The critical aspect 
here seemed to be the sense of urgency and motivation 
to make use of clinical records to improve patient care 
(Table 1). 

Where there was limited capacity at local level, leader- 
ship in relation to data orientation could be supplemented 
to a certain extent at regional level. For example, for two 
health centers located in remote communities that showed 
steady improvement (HC7 and HC8, Figure 2A), key 
informants identified competent and experienced staff in 
regional management roles, managers with an interest in 
chronic disease and in clinical and population health data 
as contributing to these patterns of change. The jurisdic- 
tion in which these two health centers (HC7 and HC8) 
were located, introduced an adult health check template 
during the period of implementation of the ABCD CQI 
project. This jurisdiction-wide strategy to improve delivery 
of preventive services was noted as one of the factors con- 
tributing to improvements in preventive care in these par- 
ticular services. Both services also received funding from 
an Australian government initiative that provided some 
support for quality improvement activities, including 
networking and coordination between services. The 
ABCD CQI project enabled health services to generate 
data for reporting for this government initiative, and the 



congruence between the project and this and other na- 
tional and State/Territory initiatives appeared to be an im- 
portant characteristic of the project in relation to 
developing shared valuing of clinical data for improve- 
ment purposes. 

The influence of macro level contexts appeared par- 
ticularly important as an explanation for change where 
CQI implementation was based primarily on local level 
initiative (CI), rather than being supported through re- 
gional systems. 

C3: Poor management, uncertainty and confusion over 
role definitions, sometimes including lack of a clear and 
consistent definition of service populations 

A key characteristic of the two health centers that 
showed persistent low performance for both diabetes 
and preventive care (HC5 and HC6, Figure 2C and 2D) 
was that they showed little interest in using clinical data 
for CQI. In explaining the low performance of these 
health centers, key informants noted that these health 
centers were managed by a central organization that had 
experienced high turnover in the CEO position, had lim- 
ited management commitment to CQI, and consequent 
delays and interruptions to CQI processes. These health 
centers were staffed by nurses and health workers, did 
not offer any on-site medical care/general practitioner 
(GP) services, and serviced transient and mobile popula- 
tions. Several other health centers that shared these add- 
itional characteristics, but were better managed and had 
greater commitment to using data for CQI, had achieved 
significant improvement in delivery of preventive ser- 
vices (though not diabetes services) through establishing 
partnerships with local GPs. 

Mechanism 2: Collective change efficacy 

Collective change efficacy relates to the belief that one's 
organization can achieve the desired change in the specific 
setting [19]. Collective change efficacy was expressed 
through program implementer explanations for outcome 
patterns that highlighted the role of networks, effective 
teamwork, and the role of intra- and inter-organizational 
learning in supporting performance improvement. 

C4: Regional or organizational infrastructure supportive 
of networking for CQI and centralization of some tasks 
and positive prior history of collaboration 

In our analysis, regional networks and their communica- 
tion systems were identified as a key factor in explaining 
patterns of change, and plausibly may have triggered the 
development of collective efficacy as a mechanism of 
performance improvement through CQI. For health 
centers demonstrating sustained high performance, 
steady improvement or marked change to improvement 
(HC11, Figure 2C and Figure 2F between years two and 
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four; and HC3, Figure 2F), reference was made to spe- 
cific communication systems used by the health board 
managing these centers, including regular cross-site 
visits and video conferencing. The health board man- 
aging this group of services encouraged the use of 
additional complementary strategies, such as a rapid 
plan-do-study-act cycle focusing on short-term care 
process targets in defined areas. Most of the health 
centers participating in such networks in our study 
were remotely located and geographically dispersed 
from one another and from other sources of support. 
These health centers had a positive prior history of col- 
laboration that went beyond project inputs and activ- 
ities. It seems plausible that remoteness and geographic 
dispersion of health centers may be an additional favor- 
able contextual factor within this broad context, possibly 
owing to the added attractiveness of networking type 
activities in counteracting structural and operational iso- 
lation of remote services. 

C5: Organizational culture unsupportive of collaboration. 
Health centers see themselves as being in competition for 
its own sake 

Where organizations were unsupportive of collaboration, 
and saw themselves as being in competition with one 
another, the mechanism of collective efficacy may have 
been inhibited. For the two health centers (both re- 
motely located) that showed persistent low performance 
in both diabetes and preventive care (HC5 and HC 6, 
Figure 2C and 2D), inappropriate reflection on perform- 
ance and early fatigue was identified as explaining this 
poor performance. 

Mechanism 3: Organizational change to encompass a 
population health orientation 

This mechanism proposes that the ABCD CQI project 
may have achieved some of its effects through its ability to 
assist organizations to develop a population health orien- 
tation. Organizations with a population-health orientation 
are characterized by: provision of population based care, 
rather than care responsive only to those presenting for 
treatment; systems thinking; working across the care con- 
tinuum from clinical prevention to palliative care and; and 
recognition that the system is primary care-led, with ef- 
fective partnering with secondary and tertiary care [20]. 
Some health centers participating in the ABCD project 
shared many of the elements of population health oriented 
organizations, and this orientation was identified as a key 
context underlying ability of these health centers to sus- 
tain high levels of performance. The further development 
of population health orientation as a mechanism for 
performance improvement through CQI was enabled by 
strong regional co-ordination and support for CQI, 



coupled with stable effective outreach workers and health 
center staff who can identify with patients and have skills 
to take broad ranging action. 

C6: Stable effective outreach workers and good regional 
co-ordination for CQI 

For some health centers demonstrating sustained high 
performance or steady improvements in diabetes and 
preventive care, their implementation of population- 
health outreach activities was cited as one of the factors 
underlying their ability to demonstrate improvements. 
For example, one health center with high performance 
in diabetes care, and marked change to improvement in 
preventive care (HC10, Figure 2C and 2F between years 
two and four) had developed a storyboard intervention 
to educate clients and community members about vari- 
ous issues of public health concern. In response to clin- 
ical audit data showing gaps in care processes for their 
service population, the health center staff extended and 
enhanced this storyboard initiative. For example, the 
health center developed a stronger proactive outreach 
approach to clinical preventive checks by including these 
as part of the work of Aboriginal Health Workers who 
used storyboards about smoking, health, nutrition, and 
hygiene done in association with clinical checks during 
outreach visits. This health center also made a concerted 
effort to access external support available for population 
health activities, including nutritional programs. Possibly 
owing to the presence of effective outreach workers and 
the support of a strong regional body, this health center 
was able to achieve these outcomes despite high turn- 
over of clinical staff. 

C7: Staff who can identify with patients and have the 
skills to take broad ranging action, including clinical 
action and action related to data system development 
and use, coupled with regional support and co-ordination 

The presence of specific staffing attributes was a favorable 
context for activation of the mechanism of organizational 
change towards a population health orientation. 

For a cluster of health centers that showed improve- 
ment or high performance in diabetes care together with 
decline in preventive care (HC12, Figure 2B and 2F be- 
tween years three and four; and HC13, Figure 2C and 2D 
between years two and four), informants noted that these 
health centers prioritized disease management ahead of 
preventive care for well adults. While health center prior- 
ities were pardy influenced by resource constraints and 
financial incentives, they were also influenced by the pri- 
orities and competencies of individual staff members. 

In explaining patterns of declining performance, fre- 
quent reference was made to key staff members who had 
left, and in some instances, not been replaced. In one of 
the examples already mentioned (HC12, Figure 2F), the 
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marked change to decline in delivery of preventive ser- 
vices, was to some extent explained by loss of a key med- 
ical staff member and a senior Aboriginal Health Worker, 
concurrent with the relocation of the center to a new 
building. However since this health center was able to 
maintain, and improve on diabetes care during this 
same period (HC12, Figure 2B), this suggests that 
greater priority was afforded to diabetes care than to 
preventive care. It may also reflect the nature of the 
services that had been delivered by the staff member 
who left, and that responsibilities had not been effect- 
ively re-allocated. While there were a number of exam- 
ples in which staffing constraints appeared to influence 
absolute levels of capacity to deliver guideline-scheduled 
services, staffing changes also appeared to influence pat- 
terns of change through changes in configurations and 
functioning of teams. 

Health centers able to sustain performance in the face 
of staff turnover were characterized by strong regional 
support systems together with commitment to good 
health center systems (HC3, Figure 2F between years 
three and four; HC12 and HC14, Figure 2B). Conversely, 
where regional support was lacking, or regional function 
was not adequate, the ability of a health center to show 
improvement was compromised. In one health center 
(HC13, Figure 2C), a slight temporary decline in diabetes 
care in otherwise consistent high performance was seen 
in a health center that was affected by a regional short- 
age of medical staff in government managed centers, 
and a restriction of services to acute care only. A delib- 
erate policy of not taking any blood samples for routine 
monitoring of chronic illness care was instituted because 
there was no medical follow up available for any abnor- 
mal results that might have been required. This regional 
crisis was associated with poorer performance in service 
delivery for this health center, but not for some of the 
others in the same region, because for this health center, 
the crisis coincided with a period of recurrent turnover 
of clinic managers. Performance in preventive care in 
this health center declined over the same period, and 
remained low, even when diabetes care returned to pre- 
viously high levels after the severe shortage of medical 
staff was relieved. 

Linkages and inter-dependence of mechanisms 

Overall, while we have identified three main mechanisms 
of change and discussed these separately, conceptually 
and practically these mechanisms are inter-linked. While 
our study was not designed to test any particular theory 
or conceptualization of organizational change, the mech- 
anisms and relationships between them had some simi- 
larities with Weiner's conceptualization of organizational 
change, which theorizes early implementation effective- 
ness [16]. In Weiner's conceptualization, 'change valence', 



or how much organizational members value change as 
being worthwhile (similar in some respects to our 
'Mechanism 1' - collective valuing of the use of data 
for improvement purposes), together with favorable in- 
formational assessment of task demands and resource 
availability, leads to collective efficacy (similar to our 
'Mechanism 2'). The inter-dependence of mechanisms 
identified in our study is also consistent with under- 
lying social cognitive theory, which for example high- 
lights reciprocal relationships between enactive mastery 
{e.g., performance feedback) and collective efficacy, par- 
ticularly where performance feedback is provided to the 
whole of the team. 

Conclusion 

Using interview data about how key informants interpreted 
outcome patterns in health centers implementing a multi- 
faceted CQI project, our study, focusing on 'middle-range 
theories', aimed to articulate more precisely the potential 
causal linkages between program activities of the ABCD 
CQI project and outcomes that were achieved. To our 
knowledge, this is the first time mechanism-based explan- 
ation has been empirically developed from a large-scale 
CQI project in primary health care. While the insights 
from the paper may be of interest to leaders of smaller 
scale CQI projects, our key messages are for wide-scale 
projects and are particularly relevant to understanding 'in- 
tegrated CQI' models in primary health care settings [5] . 

The ABCD CQI project inputs, and the mechanisms 
and contexts described are broadly similar to 'necessary 
but not sufficient' common conditions of effective QI 
reported previously [6]. Our analysis extends this and 
other broad understandings of the 'pre-conditions for suc- 
cess' to suggest how and why different conditions may 
achieve changes in delivery of care processes through both 
the design aspects of CQI (project inputs), and implemen- 
tation context. 

A particular strength of our study is its reflection on a 
key pillar of CQI — that of the use of data as a tool to 
achieve improvements in quality of care over a number 
of years of implementation, and how patterns of change 
are explained by those responsible for supporting CQI 
implementation. We recognize the possibility of con- 
firmation bias as a threat to the validity of theory- 
informed data collection, and in considering its impact 
we note the wide range of literature that informed 
the evaluative fieldwork and the ongoing emphasis 
placed on reporting what was actually observed in prac- 
tice; these aspects of the project would have minimized 
the potential effect of this type of bias. A related limita- 
tion of our study is that our data were derived from a 
relatively small number of key informants (n = 12) and, 
although these informants had detailed knowledge of the 
health centers included in the study and the process of 
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project implementation, they were not necessarily all fa- 
miliar with the range of effects operating at different 
levels of the health system. This may have influenced the 
nature of the data and the level of enquiry that was pos- 
sible. For example, although the study findings suggested 
significant effects of social structures and environments 
in explaining outcome patterns, we were not able to 
identify the specific situational mechanisms that could 
explain how social structures and environments may 
have shaped improvement leadership and management 
commitment to CQI — or influenced other aspects of 
context or mechanisms. 

Future directions for research suggested by this study 
include: further empirical inductive studies along the 
lines of the approach reported here but specifically 
designed to include a wider range of key informants and 
able to consider the whole chain of situation, action- 
formation, and transformational mechanisms [21] appro- 
priate to multi-level multi-faceted CQI interventions; 
and more focused evaluation studies of CQI that include 
testing of fine-grained realist hypotheses. 

The recommended strategies for modifying CQI pro- 
jects for greater effectiveness in different contexts flowing 
from this study include those that should (if our hypoth- 
eses are correct) make contexts more favorable to imple- 
mentation of CQI, or strengthen the action of the key 
mechanisms that we identified as explaining the patterns 
of change in our data. These strategies are consistent with 
previous literature. What our study adds, is that it identi- 
fies the context in which strategies should be applied, and 
this is based on a clear line of reasoning linking identified 
mechanisms, with program activities, context, and ob- 
served outcomes. 

Additional file 
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